Semantic indexing by augmented object association

ABSTRACT

A system for semantic indexing includes a media platform. In various embodiments, the media platform may comprise one or more processors configured to: receive a first digital object associated with a first set of semantic information; and associate the first digital object with a second digital object associated with a second set of semantic information.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority to provisional U.S. Provisional PatentApp. No. 62/280,647, filed on Jan. 19, 2016, U.S. Provisional PatentApp. No. 62/280,653, filed on Jan. 19, 2016, U.S. Provisional PatentApp. No. 62/280,671, filed on Jan. 19, 2016, U.S. Provisional PatentApp. No. 62/280,674, filed on Jan. 19, 2016, which are incorporatedherein by reference.

BACKGROUND

1. Technical Field

The various embodiments described herein are related to semanticindexing by augmented object association.

2. Related Art

Conventional media object indexing techniques are limited. For example,a library of media objects may be organized according to a treehierarchy (e.g., folders and sub-folders, or albums and sub-albums)where each node corresponds to a separate label. Membership of a mediaobject at a node in the tree (e.g., folder or album) is maintainedmanually. Thus, reclassifying media objects from one type ofclassification (e.g., location) to another (e.g., event) requiresreclassification of every media object in the library.

Keyword based classifications (e.g., textual tagging) may be betteradapted to creating and maintain dynamic albums or folders. However,keyword based classifications technique involve manual input of additionof every possible tag permutation. Thus, a multitude of tags may have tobe added for just a single location (e.g., nature, park, tree, grass).Moreover, some tags tend to be ambiguous (e.g., “Paris” could be a cityname or a person name).

Classification based on direct attributes (e.g., Apple® iPhoto®) allowsa user to create dynamic albums that are based on direct attributesassociated to the constituent media object. For example, an album caninclude all photos from a certain timeframe (e.g., from Apr. 11, 2013 toMay 5, 2014). However, direct attribute systems offer poor userinterface (UI) and impose significant limitations on searches conductedbased on direct attributes.

Limited semantic tagging (e.g., Facebook® tagging) provides dynamicclassification of media objects based on a limited set of indirectattributes. Specifically, tags do not distinguish between differenttypes of relationships that may exist with respect to the media object.For example, a person who was tagged in a photo can appear in the photo,have an interest in the content of the photo, or is a creator of thephoto.

Fully-automated media object indexing techniques are generallyinaccurate. For example, image recognition systems have only a 70%success rate at identifying even a generic object (e.g., a shoe)depicted in a media object. These systems further are not able todetermine relationships with respect to the media object (e.g., anowner, designer, and/or retailer for the shoe). By contrast, manualmethods to index media objects tend to be tedious and error prone whileoffering little user incentive.

What is needed are systems and methods for indexing media objects thatcan supports advanced searching and browsing capabilities.

SUMMARY

Systems and methods for semantic indexing by augmented objectassociation are provided. According to various embodiments, a system forsemantic indexing by augmented object association includes processorsconfigured to: receive a first digital object associated with a firstset of semantic information; and associate the first digital object witha second digital object associated with a second set of semanticinformation.

Other features and advantages of the present inventive concept should beapparent from the following description which illustrates by way ofexample aspects of the present inventive concept.

Systems and methods for curating digital objects of a digital platformare provided. According to various embodiments, a system for an objectstamping user interface includes a digital platform configured to indexdigital objects of the digital platform to identify semantic informationof each digital object, and associate a plurality of digital objectsbased on matching semantic information.

Other features and advantages of the present inventive concept should beapparent from the following description which illustrates by way ofexample aspects of the present inventive concept.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and other aspects and features of the present inventiveconcept will be more apparent by describing example embodiments withreference to the accompanying drawings, in which:

FIG. 1 is a network diagram illustrating a network environment variousembodiments;

FIG. 2 illustrates semantic indexing according to various embodiments;

FIG. 3 illustrates a stamping user interface according to variousembodiments;

FIG. 4 is a flowchart illustrating a process for stamping according tovarious embodiments;

FIG. 5 illustrates a process for adding an association to a stampaccording to various embodiments;

FIG. 6 illustrates an object selector according to various embodiments;

FIG. 7A illustrates an association selector according to variousembodiments;

FIG. 7B illustrates single and multiple association selection accordingto various embodiments;

FIG. 8 illustrates a visual interface according to various embodiments;

FIG. 9 illustrates a visual browsing menu according to variousembodiments;

FIG. 10 illustrates selection criteria according to various embodiments;

FIG. 11 illustrates a process for adding a new search selector accordingto various embodiments;

FIG. 12 illustrates a facet navigation interface according to variousembodiments;

FIG. 13 illustrates a facet display section according to variousembodiments;

FIG. 14 illustrates facet representations according to variousembodiments;

FIG. 15 illustrates a limited discrete index according to variousembodiments;

FIG. 16 illustrates a simple derived index according to variousembodiments;

FIG. 17 illustrates a fuzzy derived index according to variousembodiments;

FIG. 18 illustrates a multi-mode control switch according to variousembodiments;

FIGS. 19A-C illustrate a multi-mode UI according to various embodiments;

FIG. 20 illustrates data access computation according to variousembodiments;

FIG. 21 illustrates a process for enforcing access control according tovarious embodiments;

FIG. 22 illustrates an automatic change to access control rule statusaccording to various embodiments;

FIG. 23 illustrates a manual change to access control rule statusaccording to various embodiments;

FIG. 24 illustrates central server based synchronization according tovarious embodiments;

FIG. 25 illustrates peer-to-peer synchronization according to variousembodiments;

FIG. 26 illustrates hierarchical synchronization according to variousembodiments;

FIG. 27 illustrates access control rule cascade according to variousembodiments;

FIG. 28 illustrates a peer-to-peer browsing session according to variousembodiments;

FIG. 29 illustrates a process for initiating a peer-to-peer browsingsession according to various embodiments;

FIG. 30 illustrates a process for configuring a visual access codeaccording to various embodiments;

FIG. 31 illustrates an example lock code management interface for usewith visual access codes;

FIG. 32 illustrate an example flow for configuring visual access codesaccording to an exemplary embodiment;

FIG. 33 illustrates an example process for mapping a user identifier tophoto selection mapping according to various embodiments;

FIG. 34 illustrates an example unique user hex digest, according tovarious embodiments;

FIG. 35 illustrates an example registration process for assigning avisual access code according to an example embodiment;

FIG. 36 illustrates an example process for encoding a visual access codewith a password according to an example embodiment;

FIG. 37 illustrates an example mobile interface in accordance withvarious embodiments;

FIG. 38 illustrates an example implementation of entering a visualaccess code according to an example embodiment;

FIGS. 39A-C illustrate an example implementations of the virtual inputmethod on a website according to various embodiments;

FIG. 40 illustrates an image blending process in accordance with variousembodiments;

FIG. 41 illustrates an example implementation of hotspot positionshifting in accordance with various embodiments;

FIG. 42 is a block diagram illustrating wired or wireless systemaccording to various embodiments.

DETAILED DESCRIPTION

While certain embodiments are described, these embodiments are presentedby way of example only, and are not intended to limit the scope ofprotection. The methods and systems described herein may be embodied ina variety of other forms. Furthermore, various omissions, substitutions,and changes in the form of the example methods and systems describedherein may be made without departing from the scope of protection.

FIG. 1 is a network diagram illustrating a network environment 100according to various embodiments. Referring to FIG. 1, a user device 110communicates with a media platform 120. The user device 110 may be anydevice that is capable of communication with or causing communicationwith the media platform 120 through a wired or a wireless connection.For example, the user device 110 may be a wired or wirelesscommunication device including, for example, but not limited to, asmartphone, a wearable device, a tablet personal computer (PC), alaptop, a desktop PC, a personal entertainment system, and an embeddedprocessing system.

The user device 110 may communicate with the media platform 120 via acommunication network 130. In various embodiments, the communicationnetwork 130 represents one or more wired and/or wireless connections.For example, the communication network 130 may include, for example, butnot limited to, a wired and/or wireless local area network (LAN), awired and/or wireless wide area network (WAN), and any combinationsthereof.

The media platform 120 may be communicatively coupled with a local datastore 125. In addition, the media platform 120 may further communicatewith a plurality of remote and/or third party data sources including,for example, but not limited to, a first data source 140 and a seconddata source 150.

Association Model

Semantic Indexing

In various embodiments, the media platform 120 can associate a mediaobject with semantic information that includes, for example, but notlimited to, attributes, relationships, and classifications. The semanticinformation can be inherited from one or more other objects (i.e.,including other media objects) that each provides an individual set ofattributes, relationships, and/or classifications.

For example one media object (e.g., a photo) can depict a smiling BillGates. The media object can inherit all attributes of Bill Gates as aperson and the relationships Bill Gates has with other people (e.g.,age, an entrepreneur, an influencer, a billionaire, a philanthropist, afather, a family man, a techie, an American, etc.). Smiling implies thatBill Gates appeared in the photo and was in a good mood.

John Smith may be interested in the contents of the media object (e.g.,the photo) depicting Bill Gates. Thus, the media object can furtherinherit all attributes of John Smith as a person and the relationshipsJohn Smith has with other people although John Smith is not depicted inthe photo.

As another example, a media object (e.g., a video) can depict a vehiclebelonging to Alice after an accident. The media object can inherit allattributes of Alice's car (e.g., make, model, year, mileage, andmaintenance records), which can be determined via a third party source(e.g., Carfax®). The media object can further inherit all attributesabout Alice including, for example, but not limited to, Alice's drivingrecords, professional activities, and biographic information. Inaddition, the media object can also inherit all attributes about theparticular accident (e.g., classification as a minor or a majoraccident).

In various embodiments, the media platform 120 can allow the mediaobject to be searched via any of corresponding semantic information. Forexample, the media object depicting Bill Gates may be found through asearch for photos of 50-year old men who are smiling. Similarly, thevideo of Alice's car accident may be found through a search for videosof cars owned by women that are involved in accidents.

It is to be understood that the media object may be any kind of computerstored file including, for example, but not limited to, a text and amultimedia (e.g., photograph, video) file.

FIG. 2 illustrates semantic indexing according to various embodiments.Referring to FIG. 2, a media object (e.g., a photo of a vehicle) can beassociated with a plurality of semantic information including, forexample, but not limited to, a manual association 2 and an automaticassociation 4. In various embodiments, the media platform 120 cangenerate and add the automatic association 2 based on geolocationinformation included in the media object's meta-data. Based on themanual association 2 and the automatic association 4, the media objectcan inherit a plurality of other relationships including, for example,but not limited to, ownership (e.g., Bob Smith) and location (e.g.,Jordan Middle School parking lot).

Automatic Association Model

In various embodiments, automatic associations can be added to a mediaobject. For example, automatic associations can be generated based on ageolocation and/or timestamp. Automatic associations can also begenerated and added to the media object based on current events (e.g.,fairs, holidays, private birthdays, etc.) and weather (e.g., rain, snow,storm). In some embodiments, if the certainty of the automaticassociation is below a certain threshold, the automatic associations canbe presented to a user for confirmation.

Augmented Semantic Information

In various embodiments, the media platform 120 can add new semanticinformation to a media object, which allows the media object to besearched based on the new semantic information. For example, if BobSmith retires, then the media object depicting Bob Smith's vehicle issearchable as “photos of cars owned by retired people.”

A media object can further be searchable based on new semanticinformation that has been added to objects having existing associationswith the media object. For example, if Bob Smith's son Charlie Smith isadded as an object and Charlie Smith is a student at Jordan MiddleSchool, then the media object depicting Bob Smith's vehicle is alsosearchable as “photos of cars owned by a student's parents.”

A system for semantic indexing includes a media platform. In variousembodiments, the media platform may comprise one or more hardwareprocessors configured to: receive a first media object associated with afirst set of semantic information; and associate the first media objectwith a second media object associated with a second set of semanticinformation.

According to an example implementation, the first media object inheritsthe second set of semantic information associated with the second mediaobject. The first set and second set of semantic information can eachincludes at least one of attributes, relationships, and classifications.The one or more hardware processors can be configured to automaticallygenerate additional semantic information and associate the automaticallygenerated semantic information with the first media object. The one ormore hardware processors can be configured to automatically generate theadditional semantic information based at least in part one or more of ageolocation and a timestamp associated with the first media object.

According to another example implementation, the one or more hardwareprocessors are configured to receive additional semantic informationfrom a user and associate the additional semantic received from the userwith the first media object. For example, the user can provide theadditional semantic information at least in part by indicating anassociation between the first media object and a third media objectassociated with a third set of semantic information. In another example,the user provides the additional semantic information at least in partby indicating an association between the first media object and one ormore of an attribute, relationship, and classification.

Human-Centric Association Interface

In various embodiments, the media platform 120 presents a user interface(UI) that allows the user to quickly attach semantic information to amedia object.

Stamping User Interface

The media platform 120 allows the user to assign semantic information tomedia content using the stamping UI 300. In various embodiments, thestamping UI 300 allows the user assign semantic information to multiplemedia object using a single click. The stamping UI 300 provides a stamparea 4 displaying the categories, sources, or values of semanticinformation to be added. In one example implementation, the user clickson a target media object from a list 2, the stamp are 4 recommends alist of semantic information based on analysis of the target mediaobject. Analysis of the target media object can include objectrecognition, metadata analysis, as well as, inspecting semanticinformation of other media items associated with the target media item.The user has the option to edit the recommended list of semanticinformation in the stamp area 4 or can accept an entire set ofrecommendations. For example, a single click by the user can stamp thetarget media object with the set of semantic information shown in thestamp area 4.

The stamp area 4 can include individual associations 6, which may beadded independent of other associations. The stamp area 4 can furtherinclude association templates 8. A stamp association template 8 caninclude a group of semantic categories configured based on commonly usedassociations. Selecting a stamp association template 8 assigns a set ofsemantic information for each of the categories in the template. Forexample, a home stamp associate template can associate the target mediaobject with other media objects associated with home, add semanticinformation for the geographic location of the home, add relationshipsto people that are also part of the home category, etc. The stampassociation template 8 can be pre-configured for repeated use forcommonly used associations.

According to an example embodiment, a system for semantic indexing caninclude a media platform with one or more processing device configuredto: index media objects of the media platform to identify semanticinformation of each media object; and associate a plurality of mediaobjects based on matching semantic information. In an example, theprocessing device searches the index of media objects for semanticinformation in common with the stamp template and based on the search,the processing device sorts the corresponding media objects based oncommon semantic information; and present the sorted media objects forcuration by a user. In some embodiments, the processing device generatesa stamp template of the association comprising the common semanticinformation for applying the association to one or more media object.For example, curation can include applying the stamp template or one ormore other associations. The stamp template can be editable to modifysemantic information of the association and include multipleassociations. In some embodiments, the system includes an interface withstamp templates that allow a user to curate media objects of the mediaplatform by applying an association from a selected stamp template tomultiple media objects based on a single selection. In an example, theprocessing device receives additional semantic information from userinput to associate with one or more media object.

Stamping Workflow

FIG. 4 is a flowchart illustrating a process 400 for stamping accordingto various embodiments. Referring to FIG. 4, in various embodiments, theprocess 400 can be performed by the media platform 120.

In various embodiments, the process 400 is performed when a datacollection mode is selected. As such, the media platform 120 can collectadditional information for at least some specific associations in astamp. That is, objects with common semantic information are identifiedand the common semantic information can be grouped as an association.The groupings of common semantic information can form a template. When agrouping of semantic information is attached via the stamping action andan actual media object is defined in the system. An association can alsoform a relationship between objects with common semantic informationthat allows modifications to propagate.

For example, when the user clicks on a media object shown in a screendisplay 6, a screen display 8 appears in response to determining thatthe stamp includes a food item association. In the screen display 8, theUI allows for collecting information about the food. The screen display8 is object template type specific. After the user activates a savefeature, a new object is created (or stored) and the association isassigned to that object.

Alternately, if the data collection mode is not selected, the user canclick on a media object and other objects in the stamp area 4 areassociated with the media object.

In various embodiments, the two workflows are part of an overloadapproach of the media platform 120. The media platform 120 can collectinformation incrementally as specified by the user. Advantageously, theuser is incentivized to expend effort to input extra information becauseof an enhanced ability to search based on that information using avisual search.

Association Selection Interface

In various embodiments, the user may identify content of a media object,associate the semantic information with the media object, and select atype of the association. Once the user associates content of anothermedia object with the target media object, the target media objectsinherits the other associations of the contents of the other mediaobject. Advantageously, the user can efficiently identify and associatecontent with media object with high accuracy. FIG. 4 illustrates aflowchart for an example data collection process for associating worldobjects with media content.

FIG. 5 illustrates a process 500 for adding an association to a stampaccording to various embodiments. Referring to FIG. 5, the process 500may be performed by the media platform 120. In a screen display 2, theuser can examine media objects to be curated. From the stamp view, theuser can see the associations to be added. At the stamp view, the usercan stamp the media object by selecting (e.g., clicking, touching, etc.)a photo and all the associations in the stamp will be added to thephoto. Alternately, the user can select to add new associations to thestamp.

In response to the user clicking on a stamping mode button, the processadvances to a present a screen display 4 where the menu allows the userto select from different processes to add new associations. For example,in some embodiments, the user can add a new association by selectingobject types via the screen display 6. When the user selects an objecttype, the UI switches to screen display 8 (i.e., an object selectionscreen) that presents a searchable list of possible objects based on theselected object type. The user can search for an object to add. In someembodiments, the screen display 8 includes the stamp icon to provide foreasy navigation.

Upon selecting a specific object, an association selection screendisplay 10 is presented to allow the user to further select anassociation for the object. The user has the option to can cancel andreturn without completing the association operation. Alternately, theuser can select one or more associations on an association selectionscreen display 10 to complete the stamping operation. Subsequently oralternatively to the association selection screen 10, the UI providesadditional confirmation controls (e.g., save, cancel, etc.) to completethe association, such as depicted on a screen display 12.

FIG. 6 illustrates an object selector 600 according to variousembodiments. The user can select the object type and start a search forvarious objects to associate. The UI 600 displays the association stampthat displays existing stamps to communicate to the users the objectsare already loaded into the stamp.

Association Selection

When the user clicks the menu button to start the association editingmode, the user can select to add the association to the stamp inmultiple ways. For example, the user can select from a list of recentassociations. In another example, the user can select from a previouslyconfigured association template. Each template is a group ofassociations input by the user. For example, a template can include“baby Jim playing”, “Project X”, or “expense receipts for project Y.” Insome embodiments, the media platform 120 allows the user to define andedit a fixed number of templates. Having a limited number ofconfigurable templates allows the user to easily access preconfiguredtemplates without a template management system.

Alternately, the user can select a world object type that allows theuser to browse object types to select an association. Additionally, theuser can select from common associations, such as general associationsconfigured by an application author. For example, an application thatemploys thumbnails may use a temporary photo association.

Association Picker Flow

To streamline the association type selection between the world objectand a media object, the media platform 120 provides an optimizedassociation selector that allows the user to pick one or moreassociations. FIG. 7A illustrates an association selector 700 accordingto an example embodiment.

For example, to select one association, the user can click on theassociation button or checkbox (e.g., selection), which completes theassociation selection. To select multiple associations, the user canclick the checkbox (e.g., selection) and the association dialog boxperforms as a multiple selection dialog box. FIG. 7B illustrates singleand multiple association selection according to an example embodiment.

Association Search

The media platform 120 enables the user add associations incrementally.Moreover, the media platform 120 allows the user to quickly group theartifacts with common parameters together for faster association. Invarious embodiments, the media platform 120 links the associationprocess with the search process via the multi-mode user interface. Usingthe multi-mode user interface, the user can quickly switch betweenassociation and search and vice-versa. This quick switching allows thefollowing scenarios:

While the user is adding association, the user can switch to search modeto limit or filter the number of available media objects. As such, theuser searches on the already added semantic information or on simpleattributes such as timestamp.

While the user is searching, the user may determine that some semanticinformation is missing. Then the user can quickly switch to the stampingmode via the first level menu and start adding more associations to themedia object.

The interface enables the user to add as much information as desired inan incremental fashion, which reduces the perceived amount of work andeffort. Moreover, the user can immediately appreciate the benefits ofthe newly added information because the UI allows the user to startsearching using the new added semantic information.

Hierarchical Visual Faceted Search

In various embodiments, the media platform 120 provides a visualinterface that allows the user to quickly glance at the criteria thepresented media meet. The selection criteria can be displayed in aspecific area. In addition, the interface can have a normal mode and aminimized mode to provide more space to the user. FIG. 8 illustrates avisual interface 800 according to various embodiments. In minimizedmode, the user is in a read-only mode and cannot interact with variousselectors.

In various embodiments, presenting the query as a group of selectorssimplify the concept to the end users. Furthermore, the position of theselectors relative to each other is important and the user is able toreorder them.

FIG. 9 illustrates a visual browsing menu 900 according to variousembodiments. Referring to FIG. 9, in various embodiments, when the userclicks the central menu button in the normal mode, the media platform120 presents a first level menu view 2. This menu enables the user tobrowse the two level hierarchical facets classification. A single clickon the menu button presents the first level facet categories, whichallows the user to navigate to a second level menu view 4. In the secondlevel menu view 4, the user is provided with information about variousmedia objects. The user can determine whether multiple media objectsfrom multiple years (e.g., 2008, 2010 and 2012 but not 2011) are presentand filter to see the media objects from the month of March throughAugust. In parallel, the interface displays the selectors used to pickthe data.

Selector

FIG. 10 illustrates selection criteria 1000 according to variousembodiments. Referring to FIG. 10, the selection criteria 1000 can bebuilt by a user. As shown in FIG. 10, there can be multiple selectors 2for each facet. The selectors 2 can specify what facets values are usedfor searching. The selection criteria 1000 can include a NOT criterion 4and a MUST criterion 6, both of which can be single value selectors. Theselection criteria 1000 can further include a map location criterion 8and a value range criterion 10, both of which can be an OR criterion(i.e., at least one of the map location criterion 8 and the value rangecriterion 10 has to be true). Advantageously, the user can glancequickly and understand the criteria that is being used for the search.

Adding New Search Selector

FIG. 11 illustrates a process 1100 for adding a new search selectoraccording to various embodiments. Referring to FIG. 11, the user clicksan open menu 2, which leads the user to facets browsing mode 4. When theuse decides to search using a specific facet, the user can drag thatfacet to the selector area. The dragging action is important because itallows the user to position the selector relative to the alreadyexisting ones. Once the selector is dragged and dropped, a dialog box 6is displayed to allow the user to edit the selections in more details.The dialog box 6 may show an UI that differs from one facet type toanother. The purpose of displaying a large interface is to give thecomputer application designer more space to display various options tothe user. Furthermore, the large interface saves the user thefrustration of having to deal with small spaces. In the dialog box 6,the user can specify AND, OR and MUST criteria to different facet valuesor range of values. Once the user clicks accept, the newly addedselectors are presented in screen display 8.

Facet Navigation

FIG. 12 illustrates a facet navigation interface 1200 according tovarious embodiments.

The user can navigate between facets by selecting one of first levelfacet categories. For example, the first level of facet categories canbe divided into several (e.g., 5) main groups. The groups are specificto media objects to be searched on, e.g., who, what, when, where, andhow. When the user clicks on one of the first level facet categories,the second level facet categories can be displayed to simplify theuser's search. For example, under the “when” first level facet category,there can be multiple second level facet categories including, forexample, but not limited to, specific dates, weather conditions, eventtypes, and event names. In one exemplary embodiment, each top levelcategory has a different color to highlight and help identify thecategory.

When the user clicks on a second level facets category, the interfacedisplays the available facets and the facet values that exist in theuser's media objects. In some embodiments, a second level facetscategory can include a single facet with unlimited values.Advantageously, the facet navigation interface provides a way to quicklyglance at the media objects' meta-data. The facet navigation interfaceallows the user to explore the media collection based on themeta-information driven by curiosity. The user can be further encouragedto identify missing meta-information (e.g., the user can switch to thestamping mode and add the missing information). Moreover, the facetnavigation interface provides a unified interface for the user to startrequesting media that is not available in the media library. Forinstance, if a user discovers that the media library is missing acertain media object (e.g., a photo from the user's cousin's birthdaylast year), the user can use that interface to initiate a request forthe missing media object.

Facet Display Section

FIG. 13 illustrates a facet display section 1300 according to variousembodiments. Referring to FIG. 13, the facet display section 1300displays the facets in a particular facet category whether or not thefacets have corresponding values. By showing facets having absentvalues, the user can become aware of what is missing about thecollection of media objects the user is currently viewing. For example,the user can observes from the facet display section 1300 that neitherthe occupation facet nor the gender facet has been provided any values.

The system may display the facets in response to determining that thereare no values associated with the facets. Presenting facets with emptyvalues or no search hits communicates to the user that a backgroundsearch has determined that the facet is not useful on the data set.Empty facets may also communicate to the user that semantic data ismissing from one or more data objects or the data object is unavailable.Then the user can locate the media object to add missing the meta-dataor acquire an unavailable media object.

Facet Representation

In various embodiments, the media platform 120 displays facets to theuser in different ways based on a type associated with the facet'svalues. Advantageously, displays facets based on a type associated withthe facet's values improves communication to the user regarding theavailable types of data.

Discrete values are individual values. There are two types of discreetvalues: limited discreet values and unlimited discreet values. Limiteddiscrete values (e.g., gender) are values having a limited number ofpossible values. Facets having limited discrete values are part of asecond level facet category. By contrast, unlimited discrete values(e.g., humans, events) are displayed as an individual second level facetcategory.

Range values are values that can be grouped into a range. Some rangevalues are a continuous range which includes an unlimited number ofpossible values (e.g., timestamp). Continuous range values are alwaysdisplayed in a range or group of ranges. By contrast, an integer rangeis made up of discrete values (e.g., days of the week). Integer rangescan be displayed as groupings of discreet values. Map values can bedisplayed in a special map presentation where the map values are specialvalues.

FIG. 14 illustrates facet representations according to variousembodiments. As shown in FIG. 14, facet representations can includerepresentation of limited discreet facets 12, integer range facets 4,and unlimited discreet facets 14. The unlimited discreet facet 14 canrequire an entire subcategory to display where other facetrepresentations (e.g., the limited discreet facets 12 and the integerrange facets 4) can be grouped together.

Hierarchical Facets for Media Content

In various embodiments, facets can be organized into a two levelhierarchy. The hierarchy is stored into a data structure (e.g., tree).Each leaf node of the tree can point to an individual facet value'sindex. As such, the media platform 120 can handle different facethierarchies based on the user. For example, the facet hierarchies may beestablished based on the user level of expertise or interests.

For example, a hierarchy can include:

1) Who?

-   -   a. Name    -   b. Relationship    -   c. Interests    -   d. Profession

2) When?

-   -   a. Dates    -   b. Events    -   c. Event Types    -   d. Weather Conditions

3) Where?

-   -   a. Place Name    -   b. Map    -   c. Place Type

4) How?

-   -   a. Photographer    -   b. Camera Type    -   c. Media Type

5) What?

-   -   a. Objects    -   b. Types of Objects

Appendix B further illustrates the hierarchical faceted search engineaccording to various embodiments.

According to various embodiments, a system for performing a hierarchicalvisual faceted search for one or more media objects includes a mediaplatform. In various embodiments, the media platform may comprise one ormore hardware processors configured to: provide, to a user device, aselector user interface (UI) adapted to receive a plurality of selectioncriteria; provide, to the user device, a first level menu comprising aplurality of first level selection criteria that includes a firstselection criterion; receive, from the user device, an indication to addat least the first selection criterion to the selector UI; provide, tothe user device, a second level menu comprising a plurality of secondlevel selection criteria corresponding to the first selection criterion,wherein the plurality of second level selection criteria includes asecond selection criterion; receive, from the user device, an indicationto add at least the second selection criterion to the selector UI; addthe first selection criterion and the second selection criterion to theselector UI; and execute a search to identify one or more media objectssatisfying the first selection criterion and the second selectioncriterion based at least in part on a content of the selector UI.

In example implementations, the first selection criterion can include anidentity criterion, a location criterion, and the second selectioncriterion can include one of a name, relationship, interest, andprofession of an individual associated with a media object. For example,the first selection criterion can include a time criterion and thesecond selection criterion comprises one of a date, event, event type,and weather conditions associated with a media object. In anotherexample, the first selection criterion comprises a location criterionand the second selection criterion can include one of map coordinates,location name, and location type associated with a media object. Inanother example, the first selection criterion comprises a manner ormeans criterion, and the second selection criterion can include one of aphotographer, camera type, and media type associated with a mediaobject. In another example, the first selection criterion comprises anidentification criterion, and the second selection criterion can includeone of an object and object type associated with a media object.

Hybrid In-Memory Faceted Engine

In various embodiments, a faceted search engine may be deployed on theuser device 110. As such, the faceted search engine can rely onin-memory search indices that can be loaded on-demand based on thefacets being searched. The search indices are built on top of realobjects attribute values.

A facet is a way to search for one or more media objects. A facet can beassociated with various facet values. Moreover, each facet correspondsto a specific field in the data being indexed. Fields have data typesand expected values. To perform a search, the faceted search engine canbuild an index of all the values in each filed. Advantageously, thefaceted search engine can very quickly execute any query. By contrast,conventional faced search engines depend on pre-calculated views offixed queries and cannot handle dynamic complex queries.

In an example embodiment, a system performs a hybrid in-memory facetedsearch for one or more digital objects. The system can include a mediaplatform with one or more processing device configured to: store anindex for a data set generated using one or more indexing processes,where the index includes a mapping of values to identifiers for eachdigital object in the data set; receive an update for the index; storethe index update with a timestamp independent from the stored index; andin response to a request for the stored index, apply the index updatesto the index. In an example, the processing device determines acharacteristic associated with the data set; and the index is processesbased on the characteristic of the data set, where the index includes amapping of values to identifiers for each digital object in the dataset.

For example, the data set can include discrete data and the indexincludes multiple arrays for each digital object including at least oneof sorted values of the discrete data and identifiers. For example, theidentifiers can be grouped in ordered groups. In an example embodiment,the data set includes continuous data and each digital object is mappedto a unique timestamp.

When the mapping of values to identifiers for each digital object in thedata set includes overlapping values, the processing device candetermine a certainty factor associated with each value for each digitalobject and rank the identifier based on the certainty factor.

The request for the stored index can be a search request with querycriteria, the processing device can generate a selector object to matchquery criteria to identifiers of the index; calculate a count of theunmodified identifiers associate with each query criteria based on theselector object; and execute the search starting with the query criteriaassociated with the most unmodified identifiers of the query criteria.

Types of Indices

To index data, the faceted search engine can use different types ofindices depending on the type and nature of data being indexed. Eachindex type is built differently and has a specific purpose.

An index can be a basic index or a derived index. While each index typeis implemented in a different way, all indices have the function ofmapping real world values to certain rows of the media object they areindexing.

Basic Indices

Basic indices are directly calculated from the world object's rawvalues. Different types of basic indices may be calculated depending onthe nature of the raw value as each type of raw value is handleddifferently.

FIG. 15 illustrates a limited discrete index 1500 according to variousembodiments. Referring to FIG. 15, the limited discrete index 1500 is abasic type of index that includes discrete an unlimited. The limiteddiscrete index 1500 is kept in memory (e.g., of the user device 110) atall times and is deployed whenever the limited discrete index 1500 isneeded. In various embodiments, the limited discrete index 1500 can beimplemented as two arrays. The first array can hold the values indexedin an ascending sorting order and the starting position into the secondindex. The second array can hold the media row identifiers, which aregrouped according to the raw values and are in a sorted order in eachgroup. Advantageously, storing the limited discrete index 1500 requiresa small amount of memory.

A large discrete index includes very large number of discrete values.

A continuous index contains an unlimited number of possible values thathave almost one-to-one mapping with values (e.g., timestamp). Almostevery single photo, for example, can have a different timestamp.Continuous range values can be handled by special index structure. Forexample, continuous indices may be handled using a regular B-Treesimilar to the database indices.

A map index includes geolocation data. Geolocation data can be threedimensional data (e.g., longitude, latitude, and altitude) that arehandled as a whole. In some embodiments, map indices may be handledusing a database engine (e.g., SQLite®).

Derived Indices

Derived indices are based on other indices (e.g., basic or derived) andprovide classifications and/or implications. In various embodiments, aderived index can be a simple derived index or a fuzzy derived index.

FIG. 16 illustrates a simple derived index 1600 according to variousembodiments. Referring to FIG. 16, the index values of the simplederived index 1600 are based on values indexed by another index. Forexample, age groups may be indexed into a plurality of facet valuesincluding, for example, but not limited to, baby, toddle, kid, teenager,young adult, adult, middle age, and senior. The actual indexed valuesare derived from age raw values, e.g., a toddler is a person whose ageis between 2 and 5. In various embodiments, the simple derived index1600 is built using a single array that holds the values and thecorresponding raw value in the based index. As such, the simple derivedindex 1600 occupies very limited space in memory and can easilyaccommodate changes to the base index.

FIG. 17 illustrates a fuzzy derived index 1700 according to variousembodiments. Referring to FIG. 17, the fuzzy derived index 1700 caninclude multiple indexed values that overlap with raw values. Thus, eachmapping has a certainty factor associated with it. For example, if agegroup has a fuzzy index, then a person who is 1.8 years old can be botha baby and a toddler. However, the person is more likely to be atoddler. As such, the certainty factor for toddler is 90% while thecertainty factor for baby is 15%. The certainty factor is selected to besuitable for each value. The benefit of such fuzzy indexing is to enableus to find the same information in different way and use the certaintyto rank the search results.

Index Life Cycle

The index can be created in memory by iterating through the raw data.The index is kept in the most compact form in memory. By contrast, rawdata is accessed differently in an orthogonal operation. In order tokeep memory usage small, raw data may be iterated in batches ininstances where the volume of raw data is large. The process may includemultiple iteration passes (e.g., one pass for each index) since the datais to be loaded in a sorted order. Thus, if the volume of raw data issmall, the raw data can be loaded in to memory at once and sorted duringthe creation of the index.

Once the index is created in the most compact form, the index is savedto a file in that form. For indices that have two or more arrays, theindividual arrays are saved in the same file in a specific order, e.g.,the indexed data array before the row id array.

The index can also be updated including by adding or deleting values toand from the index. In various embodiments, if the index is already inmemory, the updates are applied to the index and the index saved to diskonce the update is complete.

If the index is on disk, then the updates are appended to an update filefor that particular index. The update file contains all the updatessorted by the timestamp. The index is not uploaded for an updateoperation. Instead of loading an index to apply one or more updates, thefaceted search engine stores the changes that are to be made to theindex. When the index is required for a search, the index is loaded tomemory from the disk, the stored updates are loaded and applied to theindex, and the index is saved to memory and can be used in a search.

Advantageously, the update process reduces unnecessary calculations thatfrom being performed each time the index is updated. The update processpreserves computational power for the index that is not being usedbetween the updates. The index is updated when the index is called(e.g., for performing a search).

Querying Model

The faceted search engine affords the user the power and control of anenterprise search interface but in an easy to understand way that doesnot have a steep learning curve. To perform a query, the user selectsone or more facet values and indicates if results must, can, or shouldnot have the selected facet values. For example, suppose the user wantsto find media objects taken during a holiday that is not Thanksgivingand depict a shoe, a dress, a bag, or sunglasses. The user can definethe following query:

MUST: event type holiday

NOT: event Thanksgiving

OR: object shoe appear

OR: object dress appear

OR: object bad appear

A user can also specify a complex query for the user's son's photoaround the house during autumn alone while my wife took the photo. Thequery can be specified as follows:

MUST: person son appear

MUST: person wife photographer

MUST: At house location

MUST: event autumn

NOT: At inside house

NOT: person any appears

Querying Mechanism

For each criteria defined by the user, the faceted search engine cancreate a selector object that operates on the facet index. Theselector's purpose is to match the criteria to the raw media objectidentifiers. It also can return the number of possible media objectidentifiers that match the given criteria, which in turn returns thepossible matches. The faceted search engine sorts the indices by thepossible matches in an ascending order. As such, the index with theleast number of matches is executed first, and the faceted search engineiterates over the list of indices and calculates the values that matchthe entire criteria.

MustSelectors[ ] CanSelectors[ ] NotSelectors[ ] mustSelectorsIdx = 0canSelectorsIdx = 0 notSelectorsIdx = 0 For each user criteria Ifcriteria is MUST MustSelectors[mustSelectorsIdx] = newSelector(criteria) mustSelectorsIdx++ else If criteria is CANCanSelectors[canSelectorsIdx] = new Selector(criteria) canSelectorsIdx++ else If criteria is NOT NotSelectors [notSelectorsIdx] = newSelector(criteria) notSelectorsIdx ++ For each MustSelectors calculatethe number of resulting row ids Sort the MustSelectors selectors by thenumber of resulting row ids in ascending order Result =MustSelectors[0].rowIds For each mustSeletor in MustSelectors<1 tomustSelectorsIdx−1>   Result = Result AND mustSeletor.rowIds For eachNotSelectors calculate number of possible row ids Sort the NotSelectorsby the number of row ids in ascending order For each notSeletor inNotSelectors <0 to notSelectorsIdx −1> Result = Result NOTnotSeletor.rowIds ORResult = CanSelectors[0].rowId For each canSeletorin CanSelectors <1 to canSelectorsIdx−1>   ORResult = ORResult ORcanSeletor.rowIds Result = Result AND ORResult

Advantageously, multiple query algorithms can be executed in parallel.Thus, each selector is independent and avoids race conditions. Theselectors are further executed in order from the strictest to the leaststrict (e.g., AND followed by NOT followed by OR). The selector orderprovides the functionality to skip the execution of the lesser strictselectors if the result set will be empty.

In addition, the algorithm may be optimized as follows:

Perform the AND and NOT part of the query using criteria that haveindices in memory If the results has row ids Perform the query on thecriteria that have unmodified indices If the results has row ids Performthe query on the remaining criteria

The above optimization reduces the need for unnecessary loading ofindices if it is know that the query result contains zero records.

Selection Operation

Selection operations are performed to select matching row identifiersfor a certain selection criteria. Each index type implements theselector in a specific way that corresponds to that particular indextype's structure.

Discrete Value Indices

Discrete value indices can be numeric or non-numeric. A numeric discretevalue index affords mathematical operations including, for example, butnot limited to, range selection, greater than, and less than.

-   -   Less Than X:        -   Binary search the facet values array and find the position            of greatest value in the index that is less than X. The            position of greatest value indicates the position in the Raw            record Ids Array. Then, the preceding row ids before the            position of greatest value position can be collected.    -   Greater Than X:        -   Binary search the facet values index and find the position            of the least value that is larger than X. The position of            the least value indicates the position in the Raw record Ids            Array. Then, the row ids after position of the least value            can be collected.    -   Equals to X:        -   Binary search the facet values index and find the position            where the value is equal to X. The position where the value            is equal indicates the position in the Raw record Ids Array.            The Raw record Ids array has the starting position. The            ending position is determined from the following index in            the facet values index.

Derived value indices can also be either numeric or non-numeric, wherenumeric derived value indices are able to support mathematicaloperations.

-   -   Less Than X:        -   Binary search the facet values array and find the position            of greatest value in the index that is less than X. The            position of greatest value indicates in the based index            values arraRaw record Ids Array. Then the row ids before            position of greatest value are collected.    -   Greater Than X:        -   Binary search the facet values index and find the position            of the least value that is larger than X. The position of            the least value indicates the position in the Raw record Ids            Array. Then the row ids after the position of the least            value position are collected.    -   Equals to X:        -   Binary search the facet values index and find the position            where the value is equal to X. The position where the value            is equal indicates the position in the Raw record Ids Array.            The Raw record Ids array has the starting position. The            ending position is taken from the following index in the            facet values index.

Query Basic Operations

Query operations (e.g., AND, NOT and OR) may be performed in multipleembodiments. For large arrays, compressed bit vector arrays are used tostore the row identifiers (e.g., roaring array) and a binary bitmask isused to perform the operations that are suitable to such data structure.For small sized arrays, the algorithms are applied such as:

The AND operation works on two arrays of row identifiers: rowIDs1 androwID2. The results are included in resultingIDs. The algorithm for anAND operation operates in linear time (i.e., O(n)) and includes:

Sort rowIDs1 and rowIDs2 in an ascending order. Using radix sort AssignrowIDs1 to the array of fewer elements Pointer1 = 0 Pointer2 = 0ResultsPointer = 0 while Pointer1< number of count rowIDs1 && Pointer2<number of count rowIDs2 if rowIDs1[Pointer1] == rowIDs2[Pointer2]resultingIDs[ResultsPointer] = rowIDs1[Pointer1] ResultsPointer++Pointer1++ Pointer2++ else if rowIDs1[Pointer1] > rowIDs2[Pointer2]Pointer2++ else Pointer1++

The NOT operation is akin to a set subtraction operation. One set of rowidentifiers are subtracted from the result. The algorithm for a NOToperation also operates in linear time (i.e., O(n)) and includes:

Sort rowIDs1 and rowIDs2 in an ascending order. Using radix sortPointer1 = 0 Pointer2 = 0 ResultsPointer = 0 while Pointer1< number ofcount rowIDs1 && Pointer2< number of count rowIDs2 if rowIDs1[Pointer1]== rowIDs2[Pointer2] Pointer1++ Pointer2++ else if rowIDs1[Pointer1] >rowIDs2[Pointer2] Pointer2++ else resultingIDs[ResultsPointer] =rowIDs1[Pointer1] ResultsPointer++ Pointer1++ while Pointer1< number ofcount rowIDs1 resultingIDs[ResultsPointer] = rowIDs1[Pointer1]ResultsPointer++ Pointer1++

The OR operation works on two arrays of row identifiers: rowIDs1 androwID2; the result is called resultingIDs. The results are included inresultingIDs. The algorithm for an AND operation operates in linear time(i.e., O(n)) and includes:

Sort rowIDs1 and rowIDs2 in an ascending order. Using radix sort AssignrowIDs1 to the array of fewer elements Pointer1 = 0 Pointer2 = 0ResultsPointer = 0 while Pointer1< number of count rowIDs1 && Pointer2<number of count rowIDs2 if rowIDs1[Pointer1] == rowIDs2[Pointer2]resultingIDs[ResultsPointer] = rowIDs1[Pointer1] ResultsPointer++Pointer1++ Pointer2++ else if rowIDs1[Pointer1] > rowIDs2[Pointer2] ifresultingIDs[ResultsPointer] < rowIDs2[Pointer2]resultingIDs[ResultsPointer] = rowIDs2[Pointer2] ResultsPointer++Pointer2++ else if resultingIDs[ResultsPointer] < rowIDs1[Pointer1]resultingIDs[ResultsPointer] = rowIDs1[Pointer1] ResultsPointer++Pointer1++

Multi-Mode User Interface

In various embodiments, the media platform 120 supports a multi-mode UIthat is adaptable to various major activity types. For example, the UIcan be in the visual search mode, the object association mode, or theobject manipulation mode. As such, the UI can focus on the majoractivity without cluttering the screen with an endless set of options.In various embodiments, the commands are positioned in substantially thesame relative place or region in the screen, allowing users access themby memory. The UI can be controlled by a mode switching control thatindicates to the user which mode the UI is in and allows the user toquickly switch to other modes.

In an example embodiment, a system for interacting with a media platformthat includes a media platform with a processing device configured to:provide, to a user device, a user interface (UI) to control a multi-modeapplication of the media platform, where the user interface comprisesmultiple mode navigation regions and each mode navigation region isassociated with a mode of the multi-mode application, where each modenavigation region comprises a specific set of functions for controllingthe mode associated with the mode navigation region. The UI is topresent the navigation region for an active mode based on detectingdevice activity, where the UI includes a global navigation menu forswitching to non-active modes and suppresses functions associated withthe non-active modes.

For example, each mode navigation region can include a specific set offunctions for controlling the mode associated with the mode navigationregion; and when the navigation region is presented, the specific set offunctions are maintained in the mode navigation region. In an exampleembodiment, the global navigation menu is controllable by a gestureinput of the user. To present the specific set of functions, the UI caninclude one or more expandable sub-mode navigation regions. The one ormore expandable sub-mode navigation regions can be presented based onthe detected device activity while the UI suppresses functionsassociated with the non-active sub-modes.

The UI can include an active mode indicator on the global navigationmenu. In some embodiments, the processing device detects deviceactivating by tracking a user's activity pattern to suggest a next modeby highlighting a shortcut on the global navigation menu. The multi-modeapplication can include at least one of a visual search mode, objectmanipulation mode, or data entry mode.

Multi-Mode Control Switch

FIG. 18 illustrates a multi-mode control switch 1800 according tovarious embodiments. Referring to FIG. 18, the multi-mode control switch1800 can be displayed on the user device 110 (e.g., a smartphone). Themulti-mode control switch 1800 displays the current mode “Visual Search”as well as additional modes the user can switch to. Alternatively theuser can use a swipe gesture (e.g., on a touch screen) to switch betweendifferent modes and/or reveal additional modes.

Multi-Mode Overview

FIGS. 19A-C illustrate a multi-mode UI according to various embodiments.Referring to FIGS. 19A-C, the multi-mode UI can be used for associatingsemantic information to media objects and for searching for mediaobjects. The stamping mode provides an interface from determiningassociations for media objects. The UI is constructed to allow efficientsearching of the media objects that can be associated in a veryeffective way that allows the user to perform batch association. Thecontrols in the stamping UI are related to association functions. Tosearch, the user can switch to the visual search mode (on the rightside). The multi-mode control switch, as described in FIG. 18, allowsfor quick navigation between modes. In search mode the visual search UIpresents control options focused on searching the media library.

Advantageously, the multi-mode UI frees the user from guessing whatactions are available for different activities (e.g., search, stamping,sharing, etc.). The multi-mode UI efficiently categorizes and presentsthe actions which are related to each activity mode. The UI commands areplaced in the same location or region of the screen for a given mode.Thus, the multi-mode UI reduces the cognitive requirements forindividual users without reducing functionality of the application.

Masking Access Control

In various embodiments, the media platform 120 controls access to storeddata objects (e.g., media objects stored in the data store 125) in amanner that does not require user accounts. By contrast, access iscontrolled based on automatic or manual data object protection rulesthat are orthogonal to user account mechanisms. Each data protectionrule selects specific data objects to be protected and can be turned onand off to make the objects inaccessible or accessible respectively.Data protection rules statuses are combinable to compute an effectivedata-masking layer. The data-masking layer determines if a given dataobject is accessible or not. Presenting data protection as simple dataselection rules simplifies complex access control mechanisms.

Data Access Computation

FIG. 20 illustrates data access computation according to variousembodiments. In various embodiments, the media platform 120 cancalculate data object visibility by performing an effective data maskingcomputation. Access control rules are combinable to create a maskingmechanism. Each rule identifies data objects that are to be inaccessible(e.g., hidden). When multiple rules are combined, the data objects thathave not been obscured by any rule are left visible to the user.

In some embodiments, the media platform 120 implements the maskingmechanism by creating a lock count and attaches the lock count to eachdata object. When an access control rule is activated, the ruleidentifies the data objects associated with the rule, determines acorresponding lock count incremented (e.g., by one), and restrictsaccess to the associated data objects. When a user performs a search,the system restricts access (e.g., hides the data objects from thesearch results, or prevents accessing the data object) to data objectsthat have a lock count greater than zero. By contrast, data objects witha lock count of zero are displayed and accessible to the user. In someembodiments, the media platform 120 can implement the masking mechanismby performing a check on whether a data object selected is associatedwith any active access control rules.

Access Control Workflow

FIG. 21 illustrates a process 2100 for enforcing access controlaccording to various embodiments.

Referring to FIG. 21, a first UI is presented to challenge the user tore-establish the user identity before editing the rules (1). A second UIis presented to enable the user to create a new access control rule oredit an existing one (2). The second UI is a dynamic UI based on theavailable meta data to pick from and the relevant tags. Moreover, thesecond UI consults with two services. First, the second UI may consultwith an ontology based tag search engine service to assist in the accesscontrol editor interface to facilitate the addition of relevant tags bythe user. For example, the user may wish to pick tags for cities inItaly and the ontology based tag search engine helps the user find themefficiently.

Second, the second UI may consult with an appropriate metadatarecommendation engine, which indicates what kind of metadata isavailable for a particular object type. For example, videos may haveduration as metadata while text documents may have a word or charactercount. The engine assists in narrowing down the selection to the userfor easy editing.

After the user confirms the final version of the access control rule,the access control rule is packaged for efficient storage andtransportation (3). The rule is made ready for execution by the mediaplatform 120. The access control rule is then stored (e.g., in a rulesdatabased with existing rules).

The effect of the rule is pre-calculated for efficient enforcement atruntime and such effect is stored with each data object (4).

Alternately, access control rules can be symbolic link access controlrules, where the rules are a simple group of hand selected data objects.In one embodiment, symbolic links are used to identify files in a filesystem to be locked. As such, when a group of symbolic links are lockedthe actual files are also locked. Data control locks implemented usingsymbolic links are separate from organizational structure.

The access control rules can also be meta-data access control ruleswhere the access control rules are based on meta-data instead of tags.

The access control rules can also be keyword based tags access controlrules where the tags are keyword matching tags and not ontology basedtags

The access control rules can also be ontology based tags access controlrules.

Changing Access Control Rule Status

Access control rule status can be changed either manually orautomatically.

FIG. 22 illustrates an automatic change to access control rule statusaccording to various embodiments. Referring to FIG. 22, an externalsystem may be working in conjunction with the masking access controlsystem. The external system is responsible for controlling which accesscontrol rules are effective which access control rules are not. Forexample, an operating system can control the access control rulesimplemented in physical storage system controller (e.g., hard drive orsolid state drive (SSD) controller). In this case, the operating systemcan add an extra level of protection that works in conjunction with theunderlying operating system level. As shown in FIG. 22, in state 1, theexternal system sends a command to the described system where thecommand instructs to turn off one or more of the rules. The systemresponds in state 2 by confirming the new status of the access controlrule after changing that rule status.

FIG. 23 illustrates a manual change to access control rule statusaccording to various embodiments. Referring to FIG. 23, in a manualmode, the user controls the status of the access control rules. When theuser wants to change the access control rule status, the system presentsa UI enabling the user to edit access control rules. For example, if theuser is challenged for authentication with a login screen to confirm theuser identity, the login screen can take a password form, a picture codeform, a lock key form, or any other authentication form. The UI can bepresented that lists the access control rules and the status (e.g.,enabled, disabled, active, deactivated, etc.) of each access controlrule. When the user clicks on one of the rules, the user is presentedwith a UI that allows the user to change the status (e.g., enabled,disabled, active, deactivated, etc.) of the rule.

According to various embodiments, a system for enforcing restrictiveaccess control with respect to a set of media objects includes on asingle device. The single device may be configured to: determine, basedat least in part on a first access control rule, to block access to atleast a first media object included in the set of media objects;determine, based at least in part on a second access control rule, toblock access to at least a second media object included in the set ofmedia objects; and provide, to a user of the single device, at least athird media object included in the set of media objects but not thefirst media object and the second media object. The device can beconfigured to provide the third media object but not the first mediaobject and the second media object based at least in part on the lockcount associated with each of the first media object, the second mediaobject, and the third media object.

Restrictive Access Control in Independent and Distributed Multi-SystemEnvironments

In some embodiments, data may be dispersed across multiple independentsystems including, for example, but not limited to, the user device 110,the data store 125, the first data source 140, and the second datasource 150. The media platform 120 can synchronize access control ruleson separate systems on a separate and a higher priority synchronizationchannel than for data synchronization. Additionally, data objectsmetadata can also use a separate synchronization mechanism and/orchannel, thereby allowing each system to enforce the rules independentlyof a centralized system.

Synchronization

In a multi-system ecosystem, different systems (e.g., the data store125, the first data source 140, and the second data source 150) arelinked together to enable the user to reuse the same access controlrules for any one system. The media platform 120 synchronizes accesscontrol rules, data objects, and metadata for data objects to enableeach system to operate independently. Multiple synchronization networkscan operate independently. For example, the access control rulessynchronization network can operate in a substantially real time fashionon a high priority level. The metadata synchronization network can alsowork on a high priority level. The data object synchronization networkmay be a third independent network.

According to an example implementation, the system can enforcerestrictive access control with respect to a set of digital objectsaccessible by a first device and second device of a user. The systemincludes the first device of the user configured to detect an updateassociated with a first system access control rule, wherein the firstsystem access control rule is to block access to at least a firstdigital object included in the set of digital objects on the firstdevice; determine, based at least in part on the update to the firstsystem access control rule, to block access to at least a second digitalobject included in the set of digital objects on a second device; andprovide, to the second device, the update associated with a first systemaccess control rule to maintain restrictive access control over the setof digital objects on a second device.

According to an example implementation, the system for enforcingrestrictive access control with respect to a set of media objectsincludes on multiple devices device for a single user. A first devicemay be configured to: determine, based at least in part on a firstaccess control rule, to block access to at least a first media objectincluded in the set of media objects; determine, based at least in parton a second access control rule, to block access to at least a secondmedia object included in the set of media objects; and provide, to auser of the first device, at least a third media object included in theset of media objects but not the first media object and the second mediaobject. The system can include a second device of the user and the firstaccess control rule can include a universal rule applicable to the firstdevice and the second device and the second access control rule caninclude a device specific rule applicable to the first device but notthe second device.

According to an example implementation, the first device and the seconddevice are configured to engage in a browsing session wherein a user ofthe second device browses the set of media objects via the first device.For example, the first device and the second device can be configured toconduct the browsing session based on a third access control rule thatis applicable to the browsing session between the first device and thesecond device for the user. The third access control rule can blockaccess to the third media object included in the set of media objects.For example, the first device can be configured to provide to the userof the second device at least the second media object but not the firstmedia object and the third media object.

Central Server

In some embodiments, synchronization can take place with a centralserver or cloud acting as a maestro. All changes are first transmittedto the central server or the cloud before the changes are propagated toother systems. FIG. 24 illustrates central server based synchronizationaccording to various embodiments.

Peer to Peer

In some embodiments, a peer-to-peer paradigm is applied in synchronizingmultiple systems. For example, peer-to-peer synchronization can useindependent versioning to track the latest updates. FIG. 25 illustratespeer-to-peer synchronization according to various embodiments.

Hierarchical Synchronization Network

In some embodiments, some systems may act as a local synchronizationserver orchestrating the status between local devices. The localsynchronization server is responsible for communication with acentralized server. For example, in the absence of a mobile network, aWiFi hotspot could host a server that orchestrates the synchronizationbetween various systems connected to the hotspot as well andcommunicates with a central server. FIG. 26 illustrates hierarchicalsynchronization according to various embodiments.

Data Access Computation

In various embodiments, data object visibility is calculated by accesscontrol rule status cascade and effecting data masking computation.

Access Control Rule Status Cascade

To support access control rule distribution and peer-to-peer browsing,multiple layers can be defined at which access control rules may beturned on or off. The layers can include, for example, but not limitedto, a universal layer (i.e., for the whole ecosystem), a system ordevice layer (i.e., for each individual device or system), anapplication layer (i.e., for systems implemented at a platform level),and a session layer (i.e., for peer-to-peer or temporary changes).

In various embodiments, access control rules can be turned on or off ateach layer. To calculate the status of each rule, rule status iscascaded from the least specific (i.e., universal) layer to the mostspecific (i.e., session) layer. The status of each rule is computed byallowing a rule status at a more general layer override the rule statusat a more specific layer. FIG. 27 illustrates access control rulecascade according to various embodiments.

Effective Data Masking Computation

The effective data masking computation is conducted in a similar manneras described above, but the process is repeated for each target session.

Peer-to-Peer Browsing Session with Access Control

In peer-to-peer browsing mode, other systems are able to browse dataobjects stored in the host device, in an ad-hoc fashion, whilemaintaining access control rules. For each satellite system that requestto browse the hosting, system can create a browsing session and changethe status of access control rules for a particular browsing session.The session rules can be included in the computation of the rule statuscascade, as described above. In some embodiments, peer-to-peer browsingconverts the host device into an ad-hoc server for the purpose of databrowsing. FIG. 28 illustrates a peer-to-peer browsing session accordingto various embodiments.

Browsing Session Initiation

FIG. 29 illustrates a process 2900 for initiating a peer-to-peerbrowsing session according to various embodiments. For a user toinitiate a peer-to-peer browsing session, the user is presented with theset of guests the user can invite to browse the user's own device (1).The user selects the desired guests then continues to examine the accesscontrol rules. The effective status of the access control rules for thenew session is presented to the user allowing the user to change theeffective status of each rule for that particular session (2). The usercan active and deactivate each access control rule for that particularguest session (3). The user is presented with a UI allowing the user tosee the currently active browsing sessions along with the guestsparticipating in each session (4). The user can add guests, removeguests, and/or terminate the session via the UI.

According to an example implementation, the system can enforcerestrictive access control for a user while browsing another user'sdevice.

Visual Access Codes

In various embodiments, access may be controlled via a visual accesscode mechanism that makes it easier for the user to remember whileproviding enhanced security by increasing the possible combinations. Thevisual access code mechanism is presented via a UI having two entryphases. The first entry phases requests the user to select one photofrom a set of photos or images, which can be preconfigured by a systemadministrator. The photo set can be the same for all users or beuser-specific. Moreover, the photo set can be the same for all devicesor be device-specific.

FIG. 30 illustrates a process 3000 for configuring a visual access codeaccording to various embodiments. The user can be presented with animage (e.g., photo) in step 1. In step 2, the user is required to select(e.g., by clicking, touching, gesturing, etc.) a subset of (e.g., four,five, etc.) hotspots from a group of possible (e.g., 16, 25, 36, etc.)hotspots. For example, the user can select a subset of 4 hotspots from16 hotspots marked on the photo by touching the hotspots in any order.In some embodiments, the image can have an overlay or marking to makethe hotspots visible and aid the user in selecting and recalling theselected hotspots. The visual access code is composed of the photo indexassociated with the photo selected at step 1 and the subset of hotspotsvalues selected at step 2. The photo index and coordinates associatedwith the subset of hotspots can be stored as an encrypted digest.

A user's visual memory is employed to store and recall the registeredimage and the hotspots by using visual cues in the image. Visual memoryis a form of memory which preserves some characteristics of our sensespertaining to visual experience. Visual memory describes therelationship between perceptual processing and the encoding, storage andretrieval of the resulting neural representations. Visual memory occursover time ranges from eye movements in order to visually navigate to apreviously visited location. Visual access codes including a subset ofhotspots on a registered image can stored longer and more readilyrecalled for providing authentication. Further, selecting trivial visualaccess codes is less likely than trivial alphanumeric passcodes (e.g.,“1111,” “1234,” “password,” etc.) since available hotspots are differentfor each photo. The user can place in memory visual information whichresembles objects, places, animals or people as a mental image of thevisual access code. The user can recall the visual access codes aspatterns from long term visual memory using different areas of theirprefrontal cortex and the anterior cingulate cortex.

According to various embodiments, a system for visual access codeprocess can include a first device configured to: present, to a user ofthe first device or service, a plurality of images; receive, from theuser, a selection of a first of the plurality of images; receive, fromthe user, a selection of at least a first of a plurality of hotspotsincluded in the first images; and generate a visual access code based atleast in part on the selection of the first images and the firsthotspot.

According to an example implementation, a registration process of thevisual access process may include a first selection from a grid ofimages (e.g., photographs, pictures, complex shapes, images, etc.) and asecond selection for a series of hotspot locations of the first selectedimage (e.g., pixel location, screen coordinates, overlay point, etc.)For example, the user may be presented with a grid of photographsdepicting various landmarks, select a picture depicting a landmark fromthe grid. Then from the selected landmark picture, the user can select aseries of hotspot locations on the selected picture. For example, theselected hotspot locations may be locations on the picture or image thatcorrespond to different parts of a landmark, the background, border ofthe picture, etc. The depicted features in the picture or image serve asvisual clues to the user where the visual cues correspond to theselected hotspot locations. Accordingly, depicted features in thepicture may be more readily stored in the user's memory than traditionalalphanumerical combinations

The system can efficiently store the user selection of the picture andset of hotspots during the registration process, as described in greaterdetail with reference to FIGS. 32-41. For example, each of the pluralityof photographs can be associated with a corresponding index number. Forexample, each index number can include a globally unique photoidentifier. In an embodiment, the first phase selection of the imagefrom a grid of images can include additional pages of grids of multipleimages. The user can scroll through multiple pages of grids of images toidentify the registered image for the first phase image selection. Eachof the images in the multiple grids of images can include an indexnumber based on the globally unique photo identifier of each image.

Each image can include a number of predefined hotspots for the user toselect a subset of for the second phase set of hotspots. According to anexample implementation, each of the plurality of hotspots is associatedwith a corresponding hotspot identifier that can be cryptographicallystored with the image index number. According to another exampleimplementation, each of the plurality of hotspots can be associated witha two dimensional coordinate of a corresponding pixel in the photograph.

According to an embodiment, the visual access code can be implemented byan authentication service, for example, on a destination device orservice. In an example, a website can replace a traditional alphanumericlogin form with a visual access code two phase input method toauthenticate the user. After the user register's a visual access codewith the authentication service, the authentication service (e.g., themobile device or website) can user various techniques for securelystoring the user's visual access code to match subsequent entry of thevisual access code to the stored registered access code. For example,after the user register's a visual access code, the authenticationservice can transform an identifier associated with the first phase andcoordinates associated with the second phase into a text string andcryptographically store the string. Then, when the user re-visits thedestination and enters the visual access code, the authenticationservice can decrypt the stored string to verify the user's authorizationto access the destination.

According to another embodiment, the visual access code can beimplemented by a client side visual authorization interface (VAI) thatreceives from the user a visual access code and outputs an alphanumericpassword to various destinations. In this embodiment, the VAI includesan algorithm that recreates the alphanumeric password based on thedestination. In an example, a user can access a website that uses atraditional alphanumeric login form with the VAI. To use the VAI withthe destination, the user employs the VAI to set-up or registers thealphanumeric password.

The visual access code system can further include a system and methodfor entering visual access codes through on-screen virtual inputmechanisms or visual authorization interface (VAI). The VAI acts as aclient side visual password entry software that does not require supportfrom other applications or websites. The VAI present to the user a userinterface that allows him to enter the password visually and then thesoftware encodes such visual access code into regular alpha-numericcharacters that are suitable for current websites and applications. Thesystem does not store the passwords anywhere and generates consistentlythem every time the user enters a visual access code.

For example, a virtual keyboard may be VAI dedicated for entering visualaccess codes. The VAI can perform client-side authentication for entryof visual passwords via the visual access code process. The VAI presentsthe user with an interface to input the visual access code independentof the device hardware. That is, the VAI provides compatibility forsecure authentication that does not require hardware, such as afingerprint reader, and maintains integrity of the visual access codesindependent of locally stored passwords.

The user may navigate to a reset my password form and launch the VAI topopulate the alphanumeric password in the destination's password form.As described below, the VAI will consistently regenerate thealphanumeric password for the destination based on the visual accesscode entered by the user. Moreover, when the same visual access code isentered into the VAI for a different destination, the VAI generates adifferent alphanumeric password. Accordingly, the VAI can authenticate auser using a visual access code compatible with the traditionaldestination login method. After the user uses the VAI to register with adestination, the destination stores the output of the VAI (e.g., analphanumeric password). The output of the VAI serves as a destinationside authentication key while the visual access code is a client sideauthentication key.

For example, after the destination records the output of the VAI, theuser can re-visits the destination, launch the VAI, enter the user'svisual access code, and the VAI will output a passcode that matches thepasscode previously stored by the destination. In various embodimentsthe output of the VAI can be based a hex digest that uses a user'sunique identifier, selection of a first image, a unique identifier ofthe first image, an image blending algorithm, selection of hotpots,shifting of hotspot coordinates, and/or one or more one-waycryptographic algorithms.

After the user registers a series or set of hotspots of a selectedimage, the user can be presented with the visual access code process forauthenticating the user to the device or service. For example, the usermay navigate to an access interface of the device or service, bepresented with multiple photographs or image during a first phase of thevisual access process. The user must recall the correct photographpreviously selected during the registered process among multiplephotographs presented. For example, the user may be presented with agrid of pictures depicting various famous landmarks. The user'sregistered image may be grouped with the same plurality of photographsfrom during the registration process or grouped with photographs thatare different from the images presented during the registration process.The user first selects the registered image from the plurality ofimages. For example, the user selected image can be matched to the indexnumber of the photograph's globally unique photo identifier.

FIG. 31 illustrates an example lock code management interface for usewith visual access codes. The lock code management interface 3100 allowsthe user to manage visual access codes, configure visual access codepreferences, assign user profiles, etc. In an example embodiment, thelock code management interface allows the user to configure differentvisual access codes based on an application category, such as mediaapplications, financial applications, work applications, etc. A masteruser can configure multiple visual access codes for different sub-usersof a service or device. For example, a parent may configure guest visualaccess codes that allow children to access gaming applications. Inanother example, a spouse may configure a partner visual access codethat allows the spouse's partner to access financial accounts but notsocial media or messaging accounts of the spouse.

FIG. 32 illustrates an example flow 3200 for configuring visual accesscodes according to an exemplary embodiment. The process begins withgathering a unique identifier from the user. The process 3200 uses aone-way cryptographic encoding to generate a consistent set of imagesfor the user, as further described in reference to FIG. 33. The workflow3200 proceeds to determine whether the user wants to set a masterpassword, as further described in reference to FIG. 36.

FIG. 33 illustrates an example process 3300 for mapping a useridentifier to photo selection mapping according to various embodiments.Process 3300 may begin with the user providing a unique identifier.Process 3300 may determine a one-way cryptographic code and generate aunique user hex digest, as described in more detail in reference toFIGS. 35-36. Then, unique user hex digest can be used to generate aunique list of photos and hotspots, as described in more detail inreference to FIGS. 40-41. Then process 3300 presents the list of photosto the user to register a passcode, as described in more detail inreference to FIGS. 37-39.

Compact Encoding

In some embodiments, the visual access code may be encoded using compactencoding. For example, in compact encoding, each photo can have an indexfrom 0 to 8 while each hotspot has an index from 0 to 15. There may beno fixed correlation between an index that is assigned to a hotspot andthe hotspot's position in the photo. The correlation between the indexthat is assigned to the hotspot and the hotspot's position in the photois photo dependent. In an embodiment, the index value associated withthe hotspot is assigned randomly. Accordingly, random index assignmentsfor the hotspot create secure access passcode.

In the example above, compact encoding creates 16 possible values. Fourof the values are selected in any order any number of times, giving riseto 3,876 possibilities. Since there are further 9 different photos, thenumber of possible combinations increases to 9×3,876=34,884, which ismore than 3 times the number of possibilities afforded by a conventional4-digit numeric passcode.

Positioned Encoding

In some embodiments, the visual access code may be generated usingpositioned encoding based on each chosen hotspot's coordinates. For thesame photo, the coordinates of each hotspot may be fixed but thosecoordinates are not transportable from one photo to another. Table 1shows how a simple hotspot index encoding as described in the previoussection is mapped to coordinate indices. For example, hotspot index 2corresponds to values (140 and 59) in Photo 1 and (89 and 147) in Photo2. With compact encoding the value 2 is shared between photos butcoordinate values for the same hotspot is not shared between photos.Moreover, the number of stored digits is also increased (e.g., 8 insteadof 4 values). Thus, positioned encoding generates even morepossibilities and renders the corresponding visual access code evenharder to break.

In some embodiment, the visual access code can depend on the size and/orresolution of the photo. For instance, in a 500×500 pixel photo, eachhotspot can generate a code from 0 to 499 in the horizontal axis andfrom 0 to 499 in the vertical axis. As such, 4 hotspots is equivalent to8 digits, which gives 1.02432860e+17 possibilities. That number isfurther multiplied by the number of photos (e.g., 9), which yields˜9e+17 possibilities for a much stronger password than a conventional8-character long alphanumeric case sensitive password with specialcharacters (i.e., 2.02095455e+11 possibilities).

TABLE 1 Photo 1 Photo 2 hotspot Index Coordinates Coordinates 0 (0, 0)(100, 100) 1 (40, 50)  (24, 135) 2 (140, 59)   (89, 147) 3 (240, 15)  (29, 225) 4 (370, 50)   (54, 135) 5 (140, 150) (214, 335) 6  (78, 150)(334, 235) 7 (67, 20) (344, 185) 8  (80, 500) (124, 195) 9  (90, 310)(249, 435) 10 (140, 240) (214, 235) 11 (400, 150) (314, 135) 12 (230,60)  (245, 135) 13 (312, 70)  (124, 235) 14 (32, 80) (274, 535) 15 (42,98) (214, 335)

Positioned Encoding with Unique Photo Identifiers

In some embodiments, every photo can be associated with a globallyunique identifier. As such, storing the passcode is dependent on thesystem or on the user. The user cannot select the same password for thetwo different systems. For example, a passcode of (0, 100, 101, 200,201, 300, 301, 400, 401) that has been encoded using positioned codingcorresponds to the first photo (i.e., photo 0). But incorporating aunique photo identifier generates a password of (38A52BE4-9352-453E-AF975C3B448652F 0, 100, 101, 200, 201, 300, 301, 400, 401), where‘38A52BE4-9352-453E-AF97-5C3B448652F0’ is the photo globally uniqueidentifier. In various embodiments, the globally unique identifier couldbe a length value (e.g., a 16 characters long number) that is hard toguess.

FIG. 34 illustrates an example unique user hex digest, according tovarious embodiments. The hex digest can include a photo selection index,a filter blending algorithm identifier, a final password mappingalgorithm identifier, photo filter bitmap, and hotspot identifiers. Forexample a unique user hex digest may begin with eight bytes designatedfor the photo selection index, followed by a one byte filter blendingalgorithm identifier, and an indicator for the final password mappingalgorithm. In an example embodiment the photo filter bitmap may consistof 27 bytes. According to an example embodiment, the unique user hexdigest can include 25 bytes for indicating hotspot shifting indices.

FIG. 35 illustrates an example registration process for assigning avisual access code according to an example embodiment. The registrationprocess 3500 may begin with a user navigating to a destination thatrequires authentication. A virtual input (e.g., VAI) method may detect atraditional login form requiring a username and password. After the usertypes a username into a traditional login form, the registration process3500 method may present a registration interface for assigning a visualaccess code to the destination using a hex digest. The user may proceedvia the VAI with selecting a first image and series of hotspots toregister a new visual access code for the destination, as describedabove.

Based on the provided username and destination identifier, the VAI canregenerate a password based on the hex digest to match a stored passwordwith the destination.

The registration process 3500 may proceed to completing the traditionallogin form with the password based on the hex digest. For example,registration process 3500 can generate the alphanumeric password usingone-way cryptographic encoding and seeding the password with adestination identifier. A password seed is used to feed the one-waycryptographic algorithm prior to generating an alphanumeric password, asdescribed in reference to FIG. 36. The seed has a direct one-to-onemapping between the user's selected hotspots and the seed. The processgenerates a password seeded with the application/website destinationname before the one-way cryptographic such that differentapplication/websites destinations have different passwords even when thesame visual passcode is used. Therefore, the user's system does not needto store the passwords for each site since the access code processconsistently replicates generation of the password based on the userinput.

According to an exemplary embodiment, the algorithm for setting up avisual access code can include

Encode the phrase with one-way cryptography i.e. sha-512 letuniqueUserHexDigest = onewayCryptography(user phrase) saveuniqueUserHexDigest to hostsystem Keystore Use the cryptography togenerate the list of photos let basePhotoIndex = getByteAtIndex(0,8,uniqueUserHexDigest) Repeat index i 0 to (number of Photos to use forkey) − 1 Let photoIndex = (basePhotoIndex + i)modulus (total number ofphotos in the system) let photo = getPhotoWithIndex(photoIndex) letblendingAlgorithmIndex= getByteAtIndex(8, uniqueUserHexDigest) letphotoFilterBitmap = getNumberOfBytesFromPosition(27,12,uniqueUserHexDigest) let photoBlendingFilter =generateFullPhotoFromBitmap(photoFilterBitmap) let finalPhoto =blendPhotoWithFilterUsingAlgorithm(photo, photoBlendingFilter,blendingAlgorithmIndex) let hotspots =getPhotoHotSpotsForIndex(photoIndex) let h=0 let shiftedHotSpots = Arrayof size of 25 for each hotspot let hotspotShift = getByteAtIndex (h+38,uniqueUserHexDigest) let adjustedHotSpot =adjustHotSpotCenterByShift(hotspots[h], hotspotShift)shiftedHotSpots.add(adjustedHotSpot)

When the destination is visited, the VAI may determine the associatedvisual access code and present the user with the virtual input method toauthenticate the user. In response to a successful visual access codeauthentication via the virtual input method, the system may populate thetraditional login form with the assigned alphanumeric password tocomplete destination authentication.

FIG. 36 illustrates an example process 3600 for encoding a visual accesscode with a password according to an example embodiment. The process3600 may input photo SHA-512 has, sorted hotspot positions by the Xaxis, and an application/website destination identifier to generate apasscode seed. The process 3600 uses the encoded passcode with a one-waycryptographic algorithm to generate the hex digest that is used to mapan alphanumeric password. In an example embodiment, the resultingpassword will generate a secure ASCII password that can include uppercase and lower case English alphabet and numbers as well as specialcharacters. According to an example embodiment the algorithm caninclude:

input passwordHexDigest let passwordBytes =getNumberOfBytesFromPosition(0,16, passwordHexDigest) let finalPassword= “” For each byte in passwordBytes If ( byte == 45 OR (byte >=48 ANDbyte <=57) OR (byte >=65 AND byte <=90) OR (byte >=97 AND byte <=122)){// take the value as is finalPassword.append(byte) continue to next byte} let modByte = byte modulus 63 if(modByte == 0){ modByte += 45 } elseif (byte >=1 AND byte <=11){ modByte += 47 } else if (byte >=12 AND byte<=37){ modByte+= 53 }else{ modByte+= 59 } finalPassword.append(modByte)

FIG. 37 illustrates an example mobile interface in accordance withvarious embodiments. To set up a visual access code for the virtualinput method, the user begins by selecting a unique passphrase such astheir name, a mother's maiden name, birthdate, or favorite location,etc. Step two the system generates a set of images for the user toselect a registration image. According to an example embodiment thealgorithm can include:

Let photoHexDigest = generateOneWayCryptographyFrom(photoBitmpa) LetselectedHotspotsXY = “” For each selected hotspot LethotspotXY=getXYForHotspot selectedHotspotsXY.append(hotspotXY) End LetsiteOrAppId = collectCurrentSiteOrAppId Let finalPasswordSeed =concat(photoHexDigest, selectedHotspotsXY, siteOrAppId) LetpasswordHexDigest = generateOneWayCryptographyFrom(finalPasswordSeed)Let passordMappingAlgorithmIndex= getByteAtIndex (9,uniqueUserHexDigest) let asciiPassword=generateAsciiPasswordWithAlgorithm(passwordHexDigest,passordMappingAlgorithmIndex)

FIG. 38 illustrates an example implementation of entering a visualaccess code according to an example embodiment. At step one, the usernavigates to a destination website or application or login screen thatrequires authentication. The VAI can detect the destination'sauthentication form and retrieves the visual access code associated withthe destinations identifier. The VAI presents the user with a set ofimages that includes the image previously registered by the user for thedestination. In response to the user selecting the image matching theregistered image, the VAI proceeds to step two to present the userhotspots of the registered image. For example, the registered image maybe of a house in include 16 possible hotspots.

In response to the user selecting a series of hotspots that matchpreviously registered hotspots, the visual input method proceeds to stepthree. For example, the user may identify four hotspots by touchingdifferent locations on the image that correspond to different parts ofthe house depicted that match the hotspots user selected during theregistration process. At step three, the virtual input method inputs thealphanumeric password stored with the visual access code into thedestination's authentication form. In some example embodiments, thevirtual input method may present a confirmation message that the userhas successfully input the visual access code. The user may proceed byclicking on the destination's authentication form to complete logging inwithout having to type an alphanumeric password.

FIGS. 39A-C illustrate an example implementations of the virtual inputmethod on a website according to various embodiments. FIG. 39Aillustrates a first phase for the virtual input method that present theuser with several images that include a previously registered image. Theuser may identify the previously registered image by clicking ortouching the registered image. In response to the user selecting theimage that matches the registered image, the user may be presented withthe registered image and instructed to identify a series of hotspots theregistered image

At FIG. 39B the user is presented a hotspot selection screen for thevirtual input method. The interface can present the user with severalhotspots for the selected image. In response to the user selecting theseries of hotspots the registered image that match the registered seriesof hotspots, the visual access code process may authenticate the user tothe device or service. For example, the user can click or touch fourlocations on the picture that correspond to the hotspot locations theuser selected during the registration process. For example, the visualaccess process determines if the user selected locations satisfy thecorresponding index numbers stored during the registration process.According to another example implementation, the visual access processdetermines if the user selected locations satisfy the two dimensionalcoordinate of a corresponding pixel in the photograph from theregistration process.

In an example implementation, to satisfy the second phase of the visualaccess code process for authenticating the user, the series of hotspotson the registered image may be identified in the same order as thehotspots were selected during the registration process. In anotherexample implementation, the series of hotspots on the registered imagemay be identified in the any order to satisfy the second phase of thevisual access code process for authenticating the user. Sincepermutations of locations and visual cues are greatly increased overtraditional alphanumerical combinations, the user first recalling thecorrect image and then identifying the series of hotspot locations onthe correct image may be sufficient for authenticating the user.

In response to the user selecting the image that does not match theregistered image, the user may be presented with a non-matching imageand instructed to identify a series of hotspots the non-matching image.To authenticate the user, the visual access code process can provide orsuppress feedback to the user regarding the first selection of an imagefrom the plurality of images. Accordingly, an unauthorized user may notbe notified whether the first selection of an image or secondidentification of the series of hotspots failed to satisfy the visualaccess code process. Repeated attempts indicating a guessing ofdifferent combinations of images from the group of images and hotspotlocations may be then be detected as a brute force attack.

If the correct hotspots are selected, the virtual input method canproceed to FIG. 39C to present the user with a confirmation and populatean alphanumeric final password in the destination's login form

To create secure visual access codes for each user, the process canemploy modified images that are unique for each user that appearvisually indiscernible. In an embodiment, the process can includeshifting center point of the original image to modify the coordinates ofthe hotspots and add a blended texture secure the image data

FIG. 40 illustrates an image blending process in accordance with variousembodiments. Blending photos for each user provides different passwordfor each user that are not detectable by simply looking observing auser's image selection. In an embodiment, the system combines a texturemasking with an original photo via a blending algorithm creates amodified photo for generating secure visual access codes. An examplephoto blending algorithm can include:

Let resultingPhoto = copyPhoto(originalPhotoSize) For each x in 0 tophotoWidth For each y in 0 to photoHeight resultingPhotoPixelAt(x,y) =resultingPhotoPixelAt(x,y) +setTransparencyTo(photoBlendingFilterPixelAt(x,y),20%) End End

In an example embodiment, the system selects a blending algorithm andthe photo filter bitmap based on data stored in the unique user hexdigest. For example, the hex digest can include value to indicate asimple overlap blending algorithm for creating the modified photo.Multiple blending and password mapping algorithms improves security ofthe access code.

FIG. 41 illustrates an example implementation of hotspot positionshifting in accordance with various embodiments. By shifting the centerpoint of the original photo coordinates, the location values of thehotspots the image can be changed to generate different passwords foreach user using the same visual image and maintaining the visualappearance of the image

An example hotspot shifting algorithm can include:

Input hotspotShiftingIndex For each hotspot hotspot.x = (hotspot.x −2) + (hotspotShiftingIndex remainder 5) hotspot.y = (hotspot.y − 2) +(hotspotShiftingIndex modulus 5) End

For example, based on the unique user phrase provided by during theregistration phase, the hex digest can provide a shift value forshifting the hotspot center and thereby differentiating the hotspotcoordinates for the image for the user. For example, an original hotspotcenter with location coordinates 30, 50 when shifted based on a shiftvalue of 9 in the unique user hex digest shifts the hotspot center tolocation coordinates 301, 49.

FIG. 42 is a block diagram illustrating wired or wireless system 550according to various embodiments. Referring to FIGS. 1 and 21, thesystem 550 may be used to implement the media platform 120. In variousembodiments, the system 550 can be a conventional personal computer,computer server, personal digital assistant, smart phone, tabletcomputer, or any other processor enabled device that is capable of wiredor wireless data communication. Other computer systems and/orarchitectures may be also used, as will be clear to those skilled in theart.

The system 550 preferably includes one or more processors, such asprocessor 560. Additional processors may be provided, such as anauxiliary processor to manage input/output, an auxiliary processor toperform floating point mathematical operations, a special-purposemicroprocessor having an architecture suitable for fast execution ofsignal processing algorithms (e.g., digital signal processor), a slaveprocessor subordinate to the main processing system (e.g., back-endprocessor), an additional microprocessor or controller for dual ormultiple processor systems, or a coprocessor. Such auxiliary processorsmay be discrete processors or may be integrated with the processor 560.

The processor 560 is preferably connected to a communication bus 555.The communication bus 555 may include a data channel for facilitatinginformation transfer between storage and other peripheral components ofthe system 550. The communication bus 555 further may provide a set ofsignals used for communication with the processor 560, including a databus, address bus, and control bus (not shown). The communication bus 555may comprise any standard or non-standard bus architecture such as, forexample, bus architectures compliant with industry standard architecture(“ISA”), extended industry standard architecture (“EISA”), Micro ChannelArchitecture (“MCA”), peripheral component interconnect (“PCI”) localbus, or standards promulgated by the Institute of Electrical andElectronics Engineers (“IEEE”) including IEEE 488 general-purposeinterface bus (“GPIB”), IEEE 696/S-100, and the like.

System 550 preferably includes a main memory 565 and may also include asecondary memory 570. The main memory 565 provides storage ofinstructions and data for programs executing on the processor 560. Themain memory 565 is typically semiconductor-based memory such as dynamicrandom access memory (“DRAM”) and/or static random access memory(“SRAM”). Other semiconductor-based memory types include, for example,synchronous dynamic random access memory (“SDRAM”), Rambus dynamicrandom access memory (“RDRAM”), ferroelectric random access memory(“FRAM”), and the like, including read only memory (“ROM”).

The secondary memory 570 may optionally include an internal memory 575and/or a removable medium 580, for example a floppy disk drive, amagnetic tape drive, a compact disc (“CD”) drive, a digital versatiledisc (“DVD”) drive, etc. The removable medium 580 is read from and/orwritten to in a well-known manner. Removable storage medium 580 may be,for example, a floppy disk, magnetic tape, CD, DVD, SD card, etc.

The removable storage medium 580 is a non-transitory computer readablemedium having stored thereon computer executable code (i.e., software)and/or data. The computer software or data stored on the removablestorage medium 580 is read into the system 550 for execution by theprocessor 560.

In alternative embodiments, the secondary memory 570 may include othersimilar means for allowing computer programs or other data orinstructions to be loaded into the system 550. Such means may include,for example, an external storage medium 595 and a communicationinterface 590. Examples of external storage medium 595 may include anexternal hard disk drive or an external optical drive, or and externalmagneto-optical drive.

Other examples of secondary memory 570 may include semiconductor-basedmemory such as programmable read-only memory (“PROM”), erasableprogrammable read-only memory (“EPROM”), electrically erasable read-onlymemory (“EEPROM”), or flash memory (block oriented memory similar toEEPROM). Also included are the removable medium 580 and a communicationinterface, which allow software and data to be transferred from anexternal storage medium 595 to the system 550.

System 550 may also include an input/output (“I/O”) interface 585. TheI/O interface 585 facilitates input from and output to external devices.For example the I/O interface 585 may receive input from a keyboard ormouse and may provide output to a display. The I/O interface 585 iscapable of facilitating input from and output to various alternativetypes of human interface and machine interface devices alike.

System 550 may also include a communication interface 590. Thecommunication interface 590 allows software and data to be transferredbetween system 550 and external devices (e.g., printers, networks,information sources, etc.). For example, computer software or executablecode may be transferred to system 550 from a network server viacommunication interface 590. Examples of communication interface 590include a modem, a network interface card (“NIC”), a wireless data card,a communications port, a PCMCIA slot and card, an infrared interface,and an IEEE 1394 fire-wire, just to name a few.

Communication interface 590 preferably implements industry promulgatedprotocol standards, such as Ethernet IEEE 802 standards, Fiber Channel,digital subscriber line (“DSL”), asynchronous digital subscriber line(“ADSL”), frame relay, asynchronous transfer mode (“ATM”), integrateddigital services network (“ISDN”), personal communications services(“PCS”), transmission control protocol/Internet protocol (“TCP/IP”),serial line Internet protocol/point to point protocol (“SLIP/PPP”), andso on, but may also implement customized or non-standard interfaceprotocols as well.

Software and data transferred via communication interface 590 aregenerally in the form of electrical communication signals 605. Theelectrical communication signals 605 are preferably provided tocommunication interface 590 via a communication channel 600. In oneembodiment, the communication channel 600 may be a wired or wirelessnetwork, or any variety of other communication links. Communicationchannel 600 carries the electrical communication signals 605 and can beimplemented using a variety of wired or wireless communication meansincluding wire or cable, fiber optics, conventional phone line, cellularphone link, wireless data communication link, radio frequency (“RF”)link, or infrared link, just to name a few.

Computer executable code (i.e., computer programs or software) is storedin the main memory 565 and/or the secondary memory 570. Computerprograms can also be received via communication interface 590 and storedin the main memory 565 and/or the secondary memory 570. Such computerprograms, when executed, enable the system 550 to perform the variousfunctions of the present invention as previously described.

In this description, the term “computer readable medium” is used torefer to any non-transitory computer readable storage media used toprovide computer executable code (e.g., software and computer programs)to the system 550. Examples of the media include main memory 565,secondary memory 570 (including internal memory 575, removable medium580, and external storage medium 595), and any peripheral devicecommunicatively coupled with communication interface 590 (including anetwork information server or other network device). Thesenon-transitory computer readable mediums are means for providingexecutable code, programming instructions, and software to the system550.

In an embodiment that is implemented using software, the software may bestored on a computer readable medium and loaded into the system 550 byway of removable medium 580, I/O interface 585, or communicationinterface 590. In such an embodiment, the software is loaded into thesystem 550 in the form of electrical communication signals 605. Thesoftware, when executed by the processor 560, preferably causes theprocessor 560 to perform the inventive features and functions previouslydescribed herein.

The system 550 also includes optional wireless communication componentsthat facilitate wireless communication over a voice and over a datanetwork. The wireless communication components comprise an antennasystem 610, a radio system 615 and a baseband system 620. In the system550, radio frequency (“RF”) signals are transmitted and received overthe air by the antenna system 610 under the management of the radiosystem 615.

In one embodiment, the antenna system 610 may comprise one or moreantennae and one or more multiplexors (not shown) that perform aswitching function to provide the antenna system 610 with transmit andreceive signal paths. In the receive path, received RF signals can becoupled from a multiplexor to a low noise amplifier (not shown) thatamplifies the received RF signal and sends the amplified signal to theradio system 615.

In alternative embodiments, the radio system 615 may comprise one ormore radios that are configured to communicate over various frequencies.In one embodiment, the radio system 615 may combine a demodulator (notshown) and modulator (not shown) in one integrated circuit (“IC”). Thedemodulator and modulator can also be separate components. In theincoming path, the demodulator strips away the RF carrier signal leavinga baseband receive audio signal, which is sent from the radio system 615to the baseband system 620.

If the received signal contains audio information, then baseband system620 decodes the signal and converts it to an analog signal. Then thesignal is amplified and sent to a speaker. The baseband system 620 alsoreceives analog audio signals from a microphone. These analog audiosignals are converted to digital signals and encoded by the basebandsystem 620. The baseband system 620 also codes the digital signals fortransmission and generates a baseband transmit audio signal that isrouted to the modulator portion of the radio system 615. The modulatormixes the baseband transmit audio signal with an RF carrier signalgenerating an RF transmit signal that is routed to the antenna systemand may pass through a power amplifier (not shown). The power amplifieramplifies the RF transmit signal and routes it to the antenna system 610where the signal is switched to the antenna port for transmission.

The baseband system 620 is also communicatively coupled with theprocessor 560. The processor 560 has access to one or more data storageareas including, for example, but not limited to, the main memory 565and the secondary memory 570. The processor 560 is preferably configuredto execute instructions (i.e., computer programs or software) that canbe stored in the main memory 565 or in the secondary memory 570.Computer programs can also be received from the baseband processor 610and stored in the main memory 565 or in the secondary memory 570, orexecuted upon receipt. Such computer programs, when executed, enable thesystem 550 to perform the various functions of the present invention aspreviously described. For example, the main memory 565 may includevarious software modules (not shown) that are executable by processor560.

Various embodiments may also be implemented primarily in hardware using,for example, components such as application specific integrated circuits(“ASICs”), or field programmable gate arrays (“FPGAs”). Implementationof a hardware state machine capable of performing the functionsdescribed herein will also be apparent to those skilled in the relevantart. Various embodiments may also be implemented using a combination ofboth hardware and software.

Furthermore, those of skill in the art will appreciate that the variousillustrative logical blocks, modules, circuits, and method stepsdescribed in connection with the above described figures and theembodiments disclosed herein can often be implemented as electronichardware, computer software, or combinations of both. To clearlyillustrate this interchangeability of hardware and software, variousillustrative components, blocks, modules, circuits, and steps have beendescribed above generally in terms of their functionality. Whether suchfunctionality is implemented as hardware or software depends upon theparticular application and design constraints imposed on the overallsystem. Skilled persons can implement the described functionality invarying ways for each particular application, but such implementationdecisions should not be interpreted as causing a departure from thescope of the invention. In addition, the grouping of functions within amodule, block, circuit or step is for ease of description. Specificfunctions or steps can be moved from one module, block or circuit toanother without departing from the invention.

Moreover, the various illustrative logical blocks, modules, and methodsdescribed in connection with the embodiments disclosed herein can beimplemented or performed with a general purpose processor, a digitalsignal processor (“DSP”), an ASIC, FPGA or other programmable logicdevice, discrete gate or transistor logic, discrete hardware components,or any combination thereof designed to perform the functions describedherein. A general-purpose processor can be a microprocessor, but in thealternative, the processor can be any processor, controller,microcontroller, or state machine. A processor can also be implementedas a combination of computing devices, for example, a combination of aDSP and a microprocessor, a plurality of microprocessors, one or moremicroprocessors in conjunction with a DSP core, or any other suchconfiguration.

Additionally, the steps of a method or algorithm described in connectionwith the embodiments disclosed herein can be embodied directly inhardware, in a software module executed by a processor, or in acombination of the two. A software module can reside in RAM memory,flash memory, ROM memory, EPROM memory, EEPROM memory, registers, harddisk, a removable disk, a CD-ROM, or any other form of storage mediumincluding a network storage medium. An exemplary storage medium can becoupled to the processor such the processor can read information from,and write information to, the storage medium. In the alternative, thestorage medium can be integral to the processor. The processor and thestorage medium can also reside in an ASIC.

The above description of the disclosed embodiments is provided to enableany person skilled in the art to make or use the invention. Variousmodifications to these embodiments will be readily apparent to thoseskilled in the art, and the generic principles described herein can beapplied to other embodiments without departing from the spirit or scopeof the invention. Thus, it is to be understood that the description anddrawings presented herein represent a presently preferred embodiment ofthe invention and are therefore representative of the subject matterwhich is broadly contemplated by the present invention. It is furtherunderstood that the scope of the present invention fully encompassesother embodiments that may become obvious to those skilled in the artand that the scope of the present invention is accordingly not limited.

What is claimed is:
 1. A system for semantic indexing, comprising: amedia platform comprising one or more processors configured to: receivea first digital object associated with a first set of semanticinformation; and associate the first digital object with a seconddigital object associated with a second set of semantic information. 2.The system of claim 1, wherein the first digital object inherits thesecond set of semantic information associated with the second digitalobject.
 3. The system of claim 1, wherein the first set and second setof semantic information each includes at least one of attributes,relationships, and classifications.
 4. The system of claim 1, whereinthe one or more processors are further configured to automaticallygenerate additional semantic information and associate the automaticallygenerated semantic information with the first digital object.
 5. Thesystem of claim 1, wherein the one or more processors are configured toautomatically generate the additional semantic information based atleast in part one or more of a geolocation and a timestamp associatedwith the first digital object.
 6. The system of claim 1, wherein the oneor more processors are further configured to receive additional semanticinformation from a user and associate the additional semantic receivedfrom the user with the first digital object.
 7. The system of claim 6,wherein the user provides the additional semantic information at leastin part by indicating an association between the first digital objectand a third digital object associated with a third set of semanticinformation.
 8. The system of claim 6, wherein the user provides theadditional semantic information at least in part by indicating anassociation between the first digital object and one or more of anattribute, relationship, and classification.
 9. The system of claim 1,the one or more processors configured to: in response to an update tothe second set of semantic information, the first digital objectinherits the updated second set of semantic information associated withthe second digital object.